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SUNNMV 


Aircrew  pcrforunce  aeasureRent  Is  a  critical  probleii  In  evaluating  the  quality  of  a  visual 
siHulatlon  systew  and  In  deteralnlng  the  effectiveness  of  aircrew  training  devices.  An  effective 
perforaance  Masureaent  systea  aust  be  able  to  separate  perforaance  Into  appropriate  coaponents 
and  describe  the  relationship  of  these  co^ionents.  This  paper  describes  a  perforaance 
aeasureaent  systea  developed  to  analyze  pilot  perforaance  In  aalntalning  altitude  In  both 
straight  and  turning  flight  as  a  function  of  the  object  density  of  the  slaulated  visual 
environaent.  The  analysis  Indicates  that  pilot  perforaance  can  be  divided  Into  perceptual  and 
task  difficulty  factors  and  that  the  effect  of  thie  visual  environaent  on  each  of  these  factors 
can  be  deteralned.  \ 


MEFACE 


This  paper  describes  a  perforMnee  aeasureaent  systeM  designed  to  quantify  pilot 
performnce  for  slaple  simlator  flight  tasks.  The  work  was  perfomed  by  the  Operations 
Training  Division  of  the  Air  Force  Huaan  Resources  Laboratory  In  support  of  the  Aircrew 
Training  Thrust  and  Aircrew  Training  Applications  Subthrust.  This  effort  was  a  part  of 
Project  2313T312,  Cognitive  Aspects  of  Flight  Training. 
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PILOT-ORIENTED  PERFORMANCE  MEASUREMENT 


1.  INTRODUCTION 

The  need  for  effective  Masuremnt  of  operator  perforunce  has  Increased  drautically  as 
■an-uch1ne  systees  have  becoae  mre  coaplex  end  costly.  Perforaance  aeasureaent  systeas  (PMSs) 
are  needed  which  will  peralt  assessaent  not  only  of  total  aan-aachine  systea  perforaance,  but 
also  of  the  coaponent  factors  contributing  to  the  total  perforaance.  To  accoapllsh  this 
assessaent.  aeasures  are  needed  that  peralt  the  decoapositlon  of  perforaance  Into  Its  perceptual, 
Inforaatlon  processing,  and  physical  control  coaponents. 

The  need  to  decoapose  overall  perforaance  Into  Its  coaponents  Is  particularly  apparent  when 
task  difficulty  Interacts  with  other  factors  to  affect  perforaance.  For  exaaple,  Rinalducci 
(1981)  exaalned  the  perforaance  of  pilots  In  aalntalning  level  flight  In  an  F-16  slaulator. 
Rinalducci  used  two  aeasures  of  perforaance:  aean  altitude  above  ground  level  (AGL)  and 
root-aean-square  (RMS)  deviation  froa  200  feet  AGL.  Both  aeasures  were  sensitive  to  the 
variables  of  visual  cues,  airspefd,  and  type  of  flight  (I.e.,  straight  or  turning).  In  addition, 
both  aeasures  were  sensitive  to  Interactions  aaong  these  variables.  One  variable,  visual  cues. 
Is  clearly  a  perceptual/ Inforaatlonal  factor.  Neither  of  the  other  two  variables  (airspeed  and 
type  of  flight)  nor  the  Interactions  are  as  aaenable  to  Intuitive  labeling.  The  perforaance 
aeasures  used  by  Rinalducci  did  not  peralt  analysis  of  the  coaponent  processes;  consequently  all 
that  can  be  shown  Is  that  these  variables  affect  perforaance,  but  not  how  they  do  so. 

Atteapts  to  decoapose  perforaance  Into  Its  coaponents  have  followed  two  general  apt^roaches. 
One  approach  uses  a  discrete  stiaulus  such  as  a  cross-wind  gust  to  elicit  a  control  Input 
(Ulerwille,  Casall,  I  Reps,  1983).  Because  the  Input  Is  elicited  by  a  discrete  stiaulus.  It  Is 
possible  to  obtain  tiaing  Inforaatlon  showing  the  contribution  of  perceptual,  subject,  and 
control  task  factors  to  the  latency  and  effectiveness  of  control  Inputs.  This  approach  provides 
Inforaatlon  not  only  about  how  well  a  pilot  controls  the  aircraft  but  also  about  the 
effectiveness  of  the  pilot's  response.  The  llaltatlon  of  this  approach  Is  that  It  can  bo  applied 
only  when  Inputs  are  aade  In  response  to  discrete  environaental  changes. 

The  second  approach  focuses  on  the  ad  lib  control  Inputs  which  operators  frequently  aake  In 
unperturbed,  steady-state  operation.  The  typical  aeasure  used  to  study  these  ad  11b  control 
Inputs  Is  slaply  the  rate  of  control  Inputs.  For  exaaple,  an  Input  rate  aeasure  which  has  been 
used  In  driving  an  autoaoblla  Is  steering  reversal  rate  (SRR),  the  rate  at  which  the  steering 
wheel  Is  reversed  through  a  snail  finite  arc.  This  aeasure  of  perforaance  Is  sensitive  to 
traffic  density  (Greenshellds,  1983),  lane  width,  speed,  and  preview  (McLean  I  Hoffnan,  1973)  and 
to  control  task  difficulty  (Hicks  A  Hlarwille,  1979).  Although  SRR  Is  sensitive  to  the  effects 
of  both  perceptual  and  task  variables.  It  has  drawbacks.  MacDonald  and  Hoffaan  (1960)  found  that 
the  addition  of  a  secondary  task  affected  SRR  differently  In  sinulated  than  In  actual  driving. 
More  Inportantly,  SRR  Is  often  uncorrelated  with  overall  steering  perforaance  (MacDonald  I 
Hoffnan,  1980). 

Slallar  control  reversal  rates  have  been  enployed  In  flight  research  to  aeasure  ad  11b 
control  Inputs.  As  In  driving  control,  such  aeasures  are  sensitive  to  flying  task  difficulty, 
but  the  reasons  for  this  sensitivity  are  not  clear.  For  exaaple,  Blonberg,  Pepler,  and  Speyer 
(1983)  used  elevator  position  reversal  rate  (EPRR)  to  aeasure  control  perforaance  In  the  A-300 
aircraft.  They  found  the  Introduction  of  an  electronic  flight  Inforaatlon  systea  (EFIS) 
Increased  EPRR,  while  other  aeasures  of  flying  perforaance  showed  the  EFIS  laproved  pilot 
perforaance.  Introduction  of  an  autopilot,  to  control  horixontal  position  and  thereby  reduce 
aircraft  task  difficulty,  caused  the  EPRR  to  decrease. 


For  Mosuros  of  ad  11b  control  Inputs  to  bo  useful.  Indices  of  Input  effectiveness,  slellar 
to  those  available  for  elicited  control  Inputs,  are  needed.  The  FNS  presented  here  Measures  both 
overall  flight  task  perforaance  and  the  effectiveness  of  ad  11b  Inputs.  Two  assuiptlons  underlie 
this  PNS:  first,  the  control  Inputs  are  elicited  by  specific  flight  conditions,  and  second,  the 
qualitative  effect  of  the  Input  reflects  the  pilot's  Intention.  That  Is,.  If  an  Input  causes  the 
aircraft  to  change  direction  of  travel,  then  the  pilot's  Intention  was  to  change  direction. 
Based  on  these  assuaptlons,  flight  control  perforaance  was  broken  down  Into  a  perceptual  task 
coaponent  and  a  physical  control  task  coaponent. 

In  this  paper,  following  a  description  of  the  PNS,  data  are  presented  to  show  the  effects  of 
perceptual  and  control  task  difficulty  on  the  perforaance  Measures.  These  data  were  gathered  In 
a  flight  slaulator  visual  data  base  evaluation,  the  results  of  which  are  presented  by  De  Nalo, 
Rinalduccl.  Brooks,  and  Brunderaan  (1983). 


II.  THE  NEASURENENT  STSTEN 

The  PNS  Must  provide  Measures  of  perforaance  that  are  sensitive  to  the  pilot's  Intentions 
aoaent  by  moaent;  therefore,  both  overall  and  coaponent  perforaance  Measures  aust  be  defined 
specifically  for  the  flying  task  considered.  The  PNS  discussed  In  the  paper  evaluated  control 
perforaance  In  Maintaining  level  flight  at  a  specified  altitude. 

Four  perforaance  Measures  were  eaployed.  Two  Measures  related  to  overall  control 
perforaance:  Target  Altitude  (TA),  defined  as  the  aean  of  the  local  altitude  alnlaa  and  aaxlaa, 
and  Altitude  Range  (AR),  defined  as  the  aean  difference  between  local  aaxlaa  and  alnlaa;  these 
Measures  give  the  altitude  the  pilot  was  atteapting  to  aalntain  and  the  degree  to  which  the 
aircraft  varied  about  that  altitude.  The  reaalnlng  two  aeasures,  Saoothness  (S)  and  Critical 
Error  Rate  (CER),  are  based  on  Individual  control  Inputs  and  were  used  to  decoapose  perforaance 
Into  Its  coaponents. 

This  PNS  eaploys  a  functional  criterion  for  defining  a  control  Input.  The  control  Inputs  of 
Interest  were  those  aade  through  the  aircraft  stick.  The  effectiveness  of  these  control  Inputs 
was  deteralned  froa  their  effect  on  the  aircraft  vertical  velocity  vector. 

Since  the  control  Inputs  that  were  exaalned  affected  the  vertical  velocity  vector,  they  can 
be  classified  In  only  two  categories:  first,  critical  control  Inputs  which  changed  the 
aircraft's  direction  ui  travel  and,  second,  noncritical  control  Inputs  that  did  not  change  the 
direction  of  travel. 

Operationally,  a  critical  control  Input  was  designated  by  a  change  In  sign  of  the  aircraft's 
vertical  acceleration.  For  exaaple.  If  the  vertical  acceleration  was  positive  (Increasing  rate 
of  cllab),  a  critical  control  Input  was  one  which  caused  the  vertical  acceleration  to  becoae 
negative  (decreasing  rate  of  cllab).  This  definition  Is  analogous  to  that  used  for  SRR.  This 
functional  criterion  aakes  the  PNS  highly  sensitive,  since  control  Inputs  are  Identified 
according  to  a  task  relevant  criterion. 

Efficient  control  would  be  expected  to  Involve  a  relatively  large  proportion  of  critical 
Inputs.  A  greater  proportion  of  noncritical  Inputs  alght  result  In  less  efficient  control  since 
Many  of  these  Inputs  do  not  result  In  error  reduction.  Based  on  this  distinction  between 
critical  and  noncritical  Inputs,  two  coaponent  perforaance  aeasures  were  defined.  One  aeasure, 
Saoothness  (S),  Is  the  proportion  of  critical  to  the  sub  of  critical  and  noncritical  control 
Inputs.  Saoothness  has  a  value  of  1.0  when  all  Inputs  are  critical  and  a  value  of  0.0  when  no 
Inputs  are  critical.  A  higher  value  of  S  represents  aore  efficient  control. 


The  other  masurc.  Critical  Error  Rate  (CER).  It  the  horizontal  distance  traveled  froa 
critical  control  Input  to  vertlccl  »cce!erat1on  sign  change  divided  by  the  tlae  froa  critical 
control  Input  to  vertical  acceleration  sign  change.  The  aore  effective  the  critical  control 
Input,  the  saaller  the  value  of  the  CER.  This  measure  reflects  the  effectiveness  of  a  critical 
control  Input  by  aeasuring  the  rate  at  which  error  continues  to  accumulate  following  the  Input. 
Effective  control  Inputs  are  those  which  result  In  low  rates  of  error  accumulation. 

These  two  aeasures,  S  and  CER,  permit  the  breakdown  of  control  performance  Into  Its 
behavioral  components.  For  this  decomposition  to  be  useful,  two  things  are  necessary:  first, 
the  component  aeasures  must  be  tied  to  psychologically  relevant  processes,  and  second,  the 
contribution  of  the  performance  components  to  overall  performance  must  be  determined.  The 
following  analysis  of  control  performance  In  a  flight  simulator  addresses  these  Issues  through 
examination  of  flying  performance  In  straight  and  turning  flight  under  varying  conditions  of 
environmental  visual  cue  quality. 


III.  SIMULATOR  FLYING  PERFORMANCE  EVALUATION 

In  the  simulator  flying  performance  evaluation,  the  effect  of  two  task  difficulty  factors  on 
the  measures  of  performance  was  examined.  One  of  the  task  difficulty  factors  addressed  was  the 
quality  of  out*of-the-cockp1t  visual  cues  provided  the  pilot.  De  Malo  et  a1.  (1903)  and  De  Halo 
and  Brooks  (1902)  have  used  the  slope,  b,  of  the  altitude  estimation  function  relating  the  actual 
sinulated  altitude  to  the  Judged  altitude  as  a  measure  of  the  altitude  cueing  effectiveness  of 
simulator  visual  environments.  The  closer  that  b  Is  to  1.0,  the  more  effective  the  visual  cues 
In  simulated  environment  are  for  estimating  altitude.  The  PMS  was  applied  to  flying  performance 
data  obtained  In  five  simulated  visual  environments,  whose  altitude  cueing  effectiveness  produced 
slopes  ranging  from  b  >.2  to  b  >.0. 

The  second  task  difficulty  factor  was  the  type  of  flight,  either  straight  or  turning.  When 
an  airplane  Is  In  wings  level  flight,  the  force  of  gravity  1$  counterbalanced  directly  by  «he 
lift  vector.  When  the  aircraft  Is  banked,  a  cosine  component  enters  the  lift  equation.  This 
cosine  component  Increases  the  difficulty  of  the  control  task  In  proportion  to  the  size  of  the 
bank  angle  up  to  90^. 

The  first  step  In  the  performance  analysis  was  to  look  at  overall  task  performance  as 
measured  by  TA.  Figure  1  shows  that,  for  both  straight  and  turning  flight,  TA  was  Inversely 
related  to  the  visual  cueing  effectiveness  as  reflected  by  the  slope  of  the  altitude  estimation 
function  for  the  different  simulator  visual  environments.  The  data  Indicated  that  performance 
improved  greatly  when  the  slope  of  the  altitude  estimation  function  exceeded  0.7.  In  addition. 
Figure  1  shows  that  turning  caused  an  Increase  In  TA  at  all  levels  of  altitude  cueing 
effectiveness.  These  data  Indicate  that  TA  Is  affected  by  both  the  quality  of  the  visual 
environment  and  the  difficulty  of  the  flight  task. 

An  understanding  of  why  pilots  raise  TA  with  Increased  task  difficulty  requires  examination 
of  another  measure  of  overall  task  performance:  Altitude  Range  (AR).  Figure  2  shows  the  effect 
of  both  altitude  cueing  effectiveness  and  task  difficulty  on  AR.  An  Inspection  of  Figure  2 
Indicates  that  AR  was  also  Inversely  related  to  altitude  cueing  effectiveness  and  task 
difficulty.  Since  AR  measures  how  precisely  the  pilot  controls  altitude,  it  affects  TA  In  that 
TA  must  be  at  least  sufficiently  large  to  preclude  collision  with  the  ground  on  minimum  altitude 
excursions. 


As  was  true  for  TA,  AR  was  sensitive  to  both  perceptual  and  task  difficulty  factors.  The 
pattern  of  these  effects,  however,  differs  substantially  for  the  two  factors.  The  effect  of  task 
difficulty  appears  to  be  quantitatively  different  on  TA  than  on  AR.  Target  altitude  Increased  by 
about  the  saae  aiount  for  both  straight  and  turning  flight  as  .altitude  cueing  effectiveness 
decreased.  Altitude  range,  however,  was  relatively  unaffected  by  the  quality  of  the  visual 
environnent  for  straight  flight  but  Increased  Markedly  for  turning  flight. 


IV.  CONPOMENT  PERFORMANCE  ANALYSIS 

Figures  1  and  2  show  that  for  straight  flight,  TA  Is  strongly  affected  by  visual  cue  quality, 
while  AR  is  relatively  constant.  At  the  saae  tlae,  turning  flight  appears  to  interact  with 
visual  cue  quality  to  Markedly  increase  AR  In  the  poor  visual  envIronMents.  In  fact,  the 
function  relating  TA  to  visual  cue  quality  In  both  straight  and  turning  flight  is  More  like  the 
AR  function  for  turns  than  for  straight  flight.  This  suggests  that  the  precision  of  the  control 
perforeance.  as  measured  by  AR,  Is  the  determinant  of  TA  In  both  turning  and  straight  flight. 
Since  turns  had  to  be  executed  during  the  flight,  the  pilot  chose  a  TA  which  permitted  an 
adequate  maneuvering  envelope  for  both  straight  flight  and  turns.  This  Implies  that  pilots 
select  an  appropriate  altitude  based  on  their  ability  to  perceive  and  control  altitude.  Uhat 
remains  to  be  shown  Is  how  the  perceptual  and  control  task  factors  act  Individually  and  in 
concert  tc  affect  control  precision.  This  analysis  Is  accomplished  by  examining  the  performance 
components,  S  and  CER. 

Smoothness  (S}  Is  a  measure  of  control  input  efficiency  since  It  measures  the  proportion  of 
inputs  that  alter  the  aircraft's  direction  of  travel.  Figure  3  shows  that  S  Is  highly  sensitive 
to  altitude  cue  quality  but  Insensitive  to  flight  task  difficulty.  Changing  the  quality  of  visual 
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Figure  3.  Median  Smoothness  for  Straight  and  Turning  Flight  as  a 
Function  of  Altitude  Cueing  Effectiveness. 


inforMtion  available  to  the  pilot  affected  the  proportion  of  critical  and  noncritical  control 
Inputs  Mde.  When  cue  quality  was  high,  twice  as  uny  Inputs  were  aade  to  change  the  direction 
of  travel  than  when  cue  quality  was  low. 


The  role  of  noncritical  control  Inputs  during  flight  Is  unclear.  At  least  two  logical 
explanations  of  these  Inputs  exist.  Since  these  Inputs  Increased  when  the  quality  of  the  visual 
envlroneent  was  poor,  they  way  serve  to  give  the  pilot  additional  perceptual  Inforaatlon  needed 
for  aircraft  control.  When  altitude  cues  are  good,  only  a  SMll  nunber  of  noncritical  Inputs  Is 
needed  to  provide  flight  control  Inforaatlon,  and  the  Majority  of  Inputs  Is  Made  to  effect  flight 
control.  When  visual  cues  are  poor.  More  noncritical  Inputs  ere  needed,  and  so  S  declines.  An 
alternative  explanation  of  noncritical  Inputs  Is  that  they  are  successive  approxleatlons  to  the 
desired  control  solution,  unfortunately,  the  available  data  do  not  perMit  separation  of  these 
two  possible  explanations. 


The  second  coMponent  perforMance  variable,  CER,  Measures  the  effectiveness  of  Individual 
critical  control  Inputs:  that  1$,  how  quickly  error  accuMulates  following  an  error  reducing 
Input.  Since  CER  Measures  the  responsiveness  of  the  Man-Machine  systOM,  It  Might  be  expected  to 
be  differentially  sensitive  to  control  task  difficulty  factors:  Figure  4  shows  this  sensitivity. 
Critical  Error  Rate  doubles  froM  about  15  ft/sec  In  straight  flight  to  about  JO  ft/sec  In  turning 
flight.  Yet  CER  does  not  vary  systOHatlcally  with  altitude  cue  quality. 


Figure  4. 


Median  Saoothnese  for  Straight  and  Turning  Flight  at  a 
Function  of  AUltodo  Cuoing  Effoetlvonoss. 


Two  coMponents  of  control  porfornanca  havo  now  been  Identified:  S,  or  Input  efficiency,  and 
CER,  or  Input  effectiveness.  Since  these  perfornance  conponents  show  the  differential 
sensitivity  to  task  difficulty  factors,  they  poralt  a  doterninatlon  of  how  Increases  In 
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difficulty  affect  the  control  process.  Increased  perceptual  task  difficulty  leads  to  a  decrease 
In  S  because  relatively  fewer  Inputs  are  effective  In  changing  the  direction  of  travel. 
Increased  control  task  difficulty,  as  reflected  In  turns,  leads  to  an  Increase  In  CER  since  the 
Inputs  are  less  effective  In  altering  the  direction  of  flight. 

Conceptually,  the  effects  of  variation  In  control  efficiency  (S)  and  Input  effectiveness 
(CER)  can  be  related  to  overall  control  perforaance  as  reflected  In  AR.  When  Input  efficiency 
decreases,  cue  to  Increased  perceptual  task  difficulty,  directional  changes  occur  less 
frequently,  and  AR  Increases.  Slallarly.  when  Inputs  are  less  effective  due  to  Increased  control 
task  difficulty,  the  aircraft  responds  sore  slowly,  and  AR  again  Increases.  Therefore,  AR  Is 
directly  affected  by  both  the  quality  of  the  visual  envlronaent  and  the  difficulty  of  the  flight 
control  task. 


V.  DISCUSSION 

The  four  perforwance  aeasures  described  In  this  paper  break  down  flight  control  perforwance 
into  coBponent  processes.  Two  of  these  Measures.  TA  and  S,  were  priaarlly  influenced  by  the 
altitude  cueing  effectiveness  of  the  visual  envlronaent.  The  reaelnlng  two  Measures,  AR  and  CER, 
were  affected  acre  by  the  difficulty  of  the  flight  control  task  than  by  the  altitude  cueing 
effectiveness  of  the  visual  scene.  Taken  together,  these  four  Measures  describe  flight  control 
perfornance  on  the  basis  of  both  perceptual  and  task  difficulty  coMponents. 

In  addition,  the  results  of  this  Investigation  Indicate  that  a  perfornance  aeasureaent  systeM 
requires  an  analysis  of  both  overall  flight  porforaance  and  control  Inputs.  Target  altitude  and 
AR  represent  whoHstlc  Measures  based  on  the  aircraft's  position  In  space.  Snoothness  and  CER 
reflect  the  specific  control  Inputs  aade  by  the  pilot  during  flight.  Both  types  of  Measures  are 
necessary  for  adequately  understanding  pilot  control  perfornance. 
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